Jesse Ball is trying to show me how to lucid-dream. The 38-year-old writer and I are in a café in Lincoln Park to discuss his novels and poetry. Almost immediately we find ourselves talking instead about the class on dreaming he teaches at the School of the Art Institute of Chicago.

“You can begin by looking at a book. When you’re dreaming, you can pick up a book and then put it down and pick it up again, and the title usually changes. Usually the dream can’t keep up.” Here he picks up a magazine that happens to be lying on a nearby table, holds it up, then sets it down again. “You can constantly do this during the day, these escape tests, to know if you’re dreaming. Dreams are an accumulation of habits. What are you but your habits? And these habits continue in dreams. So when you’re awake, you can pick up a book—or you could do this when you are walking or when you go through a doorway—and ask yourself, Is this a dream? You have to get into the habit of it. Eventually, when you’re actually asleep, you’ll pick up a book and ask, Is this a dream? and the answer will be yes. And then you can fly around.”

It’s strange stuff, this notion of training yourself to control your dreams, and it could come straight from one of Ball’s surreal novels, which have made him into one of the most acclaimed experimental writers to come out of Chicago in years. Ball, who was just named a Guggenheim fellow, creates worlds that exist somewhere between the known and the unknown, the real and the absurd. His work has garnered attention from the likes of The Paris Review and the National Endowment for the Arts for its ingenious depictions of well-intentioned characters who must negotiate difficult mazes of tragedy and the unending tangles of unjust social systems.

“Books should have a purpose,” Ball says. “Books should be practical in some sense. Because [in life] you are going to be racked with grief and afflicted by mortal things, horrific things that you can’t even predict. Everything that’s good that’s given to you is going to be taken away, so you have to learn how to behave and how to navigate these things.”

Ball’s work is as spellbinding as it is daring. His 2014 novel, Silence Once Begun, about a Japanese man tricked into confessing a crime he did not commit, is presented as a series of interviews with witnesses. A Cure for Suicide, from 2015, which centers on a nameless man who has lost his memory, is told mostly through dialogue. His new novel, How to Set a Fire and Why, out in July, takes the form of the diary of a troubled teenage girl named Lucia.

“Each of the books is quite unlike the other ones,” says Ball, whose chin and cheeks are dotted with a range of whiskery growth. “There’s a sentiment or feeling I’m trying to get at. The form of the book is the form that allows me to do that.” What connects each of these books is his pursuit of a single act of discovery, the moment when a character realizes the world is much more complicated than he or she thought. Publishers Weekly notes in a starred review for his latest: “Lucia details a philosophy that smartly parallels the novel’s own—namely, that writing literature is, like arson, an act of creation and destruction.”

Ball’s interest in the liminal nature of words began when he was a boy. His father, Robert, handed him Grendel, by John Gardner, a modern retelling of the Old English poem Beowulf, written from the monster’s point of view. Rather than being daunted by the heavy topics of good and evil and the necessity of myths, Ball was fascinated. Having been brought up as an atheist, he learned to question the way the world worked. His father, when asked of his political beliefs, would reply that he was a Peter Kropotkin anarchist, a reference to the 19th-­century Russian scientist who advocated decentralizing governments throughout Europe. “I guess there was also some obscure religious movement in England called the Fifth Monarchists,” Ball says, “who believed the British king was the Antichrist.” His father identified with them too.

Ball grew up in Port Jefferson, New York, on Long Island. His mother, Catherine, was a librarian; his father was an administrator for Medicaid. Before that, his father had been a seminarian, and his mother had trained to be a nun before deciding the religious life wasn’t for her. The two met at a bookstore in Washington, D.C.

As a child, Ball spent a significant amount of time in intensive care units with his older brother, Abram, who was born with Down syndrome and suffered from complications related to it. Books became a refuge. “Sometimes I’d read. There was the humming of machines, his ventilator. He couldn’t really communicate very much, but he could smile, demonstrate emotion.”

When Ball was 12, he began writing poems in secret. He had no interest in showing his work to anybody. “There’s a phase where a writer or any kind of artist has to focus on just doing the work for themselves.” His father died when Ball was 18; his brother, three years later. Ball doesn’t say much about those early losses, but they clearly imbue his writing with an emotional gravity few writers his age carry.

He continued to write for himself at Vassar College and eventually headed to Columbia University for an MFA. There he met poet Richard Howard, who helped him publish his first book of poems, March Book, when he was 24. He published his first novel, Samedi the Deafness, a few years later, in 2007, to promising reviews.

That same year, Ball took a teaching position at the School of the Art Institute of Chicago. He found the city “very American”: “New York feels like sometimes it’s not part of the United States. So does L.A. Chicago feels like it’s a big city that’s part of America.” Initially, he says, he was “conflicted about teaching, what it would mean to teach.” Unlike most creative writing instructors, who lecture on issues of craft or content, Ball decided to focus instead on cultivating the young writers he worked with as people with distinct interests. “I thought if the students would become more interesting [people], they would become more interesting writers.”

So he developed a number of innovative classes, including the one on lucid dreaming. In the spirit of writers like Charles Baudelaire, who believed traversing a city was its own kind of art form, Ball created a class on walking. He also developed one on lying. “Lying is our stock-in-trade as social creatures,” says Ball. “The class is not about judging if someone is lying or not lying; it’s more about navigating this morass of lies without judgment. As a writer of fiction, lying is the central thing to all books.”

Over the next several years, Ball honed his own writing, moving from lucid, fable-like poetry to dreamlike prose, and developed angular books grounded in a postmodern sensibility. Though Ball, who lives in Lincoln Park with his dog, Goose, doesn’t talk much about his personal life—he’s been married twice—one of the most engaging aspects of his work is his ability to depict the difficulty of moving through a world of complex and mysterious people.

James McManus, a colleague of Ball’s in the SAIC writing program, says: “Of all the experimental writers, he is the most readable—he still gives you plot, suspense, and an emotional wallop. He’s our biggest draw in the program; people move to Chicago to work with him.”

Silence Once Begun, which was a finalist for the Young Lions fiction award and the Los Angeles Times Book Prize, propelled Ball into the national literary spotlight in 2014. The next year, A Cure for Suicide was longlisted for the National Book Award. But Ball doesn’t dwell on the accolades. “You can’t build a life or a career around things that don’t happen very often.”

For his new novel, How to Set a Fire and Why, Ball pivots to the tale of a teenage girl struggling to make her way through the death of her father, her mother’s descent into mental illness, and the confining world of high school. Eventually, she falls in with a group of young arsonists. “I wanted to write about this young woman, this school—many of these details have to do with the contemporary world,” says Ball. “In other books, the characters are able to avoid those details, but when you’re in high school, you cannot. You are constantly afflicted by them.”

Jennifer Jackson, who has edited all of Ball’s novels at both Vintage Books and Pantheon, believes that his writing has become “more character driven, more emotionally resonant”: “His older books were a bit more intellectual,” says Jackson, “and he has shifted to books that completely break your heart.”

Walking along Cortland Street from his studio workspace in a seemingly abandoned industrial building, Ball and I discuss the way he put Fire together. He explains that he needed to find a place where he could be anonymous, where he could delve into Lucia’s world and escape his own. After roaming the city, he discovered the now-shuttered Ch’ava Cafe on North Clark Street. There he completed the book in one week, always trying to move the characters and story forward, rewriting as little as possible.

As we walk, Ball mentions Russian bibliophiles who used to handwrite or type up the books they admired to pass on to loved ones or friends. He riffs on the idea of a world where writers are paid a stipend and all worries about commercial success are removed from artistic creation. He talks about his next projects. One is a manual on lucid dreaming for children. But it’s the process of writing itself that matters most to him: “The books are something that just appear at transitions. They’re not the point. The making of something is just a point where I discover what I think.”

We keep talking. A low fog hangs in the air as we cross a bridge, the river below green and dark. To an outsider, a children’s book might appear to be a peculiar choice, but for Ball it seems like a fitting one.