09/21/2022

Developing a thesis question Humans have always been creating tools, and technological innovation is constantly automating new aspects of human labour and shifting our culture. Advancements in printing transformed the way people communicated and understood the world. This increased more as we started using the Internet. At the foundation of this is typography, often the basis of which we communicate and share ideas.

How can we build tools for typeface design that empower the individual to shape the future of communication?

Research areas My main research areas are technology and culture in typeface design, and machine learning and techniques for generative typography. Some questions I have are – how do our tools impact us? How does automation affect us? What does this mean for globalization and languages? Are tools creating design standardization? Is this good or bad? What are some of the dangers and benefits? What does this mean in the digital information age?

Goals First, I want to contribute to the future of tooling by introducing a new approach to forming type, building a use case for existing research, and making people think about the tools we use.

Second, promote the craft and bring more awareness to type design. I want to lower the barrier of entry to type design by showing people that making letters can be fun and not scary. Grow the pie.

Third, empower people to make their own tools. Type is also a tool. Make it easier for people to shape their own culture and communication, digital or not. For the people who like to make things themselves, for the people who care about this kind of stuff... not everyone, and that's OK. There are people who make their own fonts, and people who use others'. People who make their own playlists, and people who listen to others'.

Outcomes I plan to conduct and share my research findings on the cultural effects of automation and standardization in tooling, what is possible in generative type with current research discoveries, experiment with GANs, and books I read.

I would also like to develop a proof-of-concept Figma plugin for making fonts. I chose Figma because there's an existing user base, meaning it's more discoverable and you don't have to learn another tool. The future of digital tooling is cloud-based and collaborative. It's hard to build these things on your own, and typical typeface design tool don't have these features. By building on Figma, you have these by default. Plus, you don't have to pay for hosting or build some functions from scratch due to the Figma API.

​ 10/26/2022

My approach to design is iterative, in which ideas are conceptualized and tested, and built upon, based on feedback. I started off with a broad interest in combining typeface design and machine learning. After sharing my thoughts with the thesis class, surveying the current landscape of machine learning for vector graphics and tools for typeface design, I initially wanted to work towards building a Figma plugin that would generate an entire character set based on a single glyph.

I tested this idea by talking to Arshia Sobhan, a former Digital Futures graduate student who completed his thesis on generative Islamic calligraphy. Our conversation shed light to the possibilities of collecting data using different mediums, and creating more interesting typefaces that do not solely rely on existing font files.

Reading often changes the way I view my project while I am working on it. I am learning how to make tools as I explore what it means to make tools, and how our tools shape our work. While I examine code repositories on neural networks for vector graphics and watch videos on linear regression, I am in the middle of books that question the role of technology in society, and have Figma’s plugin API documentation open on another tab. I keep an eye out for current events and play with new generative tools, that show me new aspects of what is possible with technology.

​ 12/08/2022

Demo I built a "Text Decorator" tool with with Python, Google Colab, Gradio, Hugging Face Spaces, Stable Diffusion, PyTorch, and PIL. Using my tool, I created a series of letters from A-Z, titled AI Alphabet, which will be at the DF Open Show on December 13.

Readings This semester, I finished reading The Medium is the Massage, The Information Bomb, and Cryptography: A Very Short Introduction. I am still reading Caps Lock: How Capitalism Took Hold of Graphic Design, and How to Escape From It, The Dream Machine, and Who Owns the Future? I am also reading multiple academic papers to to learn about what is going on in the field of AI research, vector graphics, and fonts – including the papers for DeepSVG, CLIPDraw, GlyphGAN, DeepVecFont, and more. Hoping I can get a better understanding of different models and how to incorporate methods into my design practice.

Conversations My key takeaways from consultations and critiques this fall are, first, to think about what the project means to me personally. Second, "standardization" can have different meanings to different people. And third, with the invention of printing, and now digital technology, many aspects of the spoken word are lost in translation, such as silence.

From talking to Arsh, I realized that there are more interesting ways of getting data other than searching the Internet for fonts. And preparing a dataset is important, although it can be time consuming, it will save you a lot of time later on, when it comes to training your model.

Other people who are knowledgeable in machine learning have advised me that it's difficult to translate papers into code, but if you reach out to authors, they are usually happy to talk about their research. Although it may not seem related to your project, learning ML fundamentals such as linear regression and gradient descent are important and will be applicable later on. Also, if you are interested in text to SVG, it may be useful to first try to generate descriptions of an SVG.

Next semester I can work on adding other features to Text Decorator, such as custom fonts, colours, and different models. I would like to explore the use of APIs in type design, with NLP and audio. I would also like to replicate machine learning experiments, particularly glyph style transfer. I will also read more books and academic papers. ​

01/19/2023 Audio adaptive typography driven by machine learning Communicating your ideas and feelings is a big part of life. From birth, we learn to express our thoughts through things like words, actions, and pictures. And we develop our own ways of talking to the world. One way I do this is through fonts. Sometimes I use Times New Roman. Other times, I don’t. Whether the reader realizes it or not, they are interpreting my font choices.

Music is another form of communication. Many people have written songs that sum up how I feel, better than I can express in my own words. It’s a nice feeling when someone likes your playlist, or listens to your composition. Kind of like relating in a different level. This made me think, "How can fonts be used to communicate the feelings of listening to a song in real-time?"

I plan to make a variable font that reacts dynamically to music. How would a song manifest itself as a font? I imagine that different genres, tempos, and lyrical content would translate to typographical characteristics.

How it started I have always liked picking fonts for projects, and recognizing fonts being used in public. But it wasn’t until a year ago, in January 2021, when I made my own font for the first time. As someone who designs software for a living, I was curious about approaches to typeface design outside of traditional graphic design methods. I tested out Figma for drawing characters, and experimented with training a GAN on a logo dataset.

Later in the year, I began to pay closer attention to cinematic title screens and typography for music videos. Around the same time, I had been looking at Spotify’s API documentation, out of curiosity for the different endpoints available to analyze a listener’s music taste. The research I had done prior to and during Thesis 1 allowed me to combine several of my interests into one project.

Assembling my toolkit Last semester, I spent Thesis 1 investigating the question, "How can we build tools for typeface design that empower the individual to shape the future of communication?" The invention of printing completely changed how the world communicated. People exchanged ideas, spread beliefs, and advanced knowledge at a much faster pace. Today, the Internet has made it even easier for us to share information with each other, bringing geographically distant communities to the computer screens in our very own homes.

As containers for written language evolve, so do writing systems and their techniques of production. Type design began as a physical practice – from the 16th to 18th century, it was the individual who both designed type and made it. However, in the late 1700s, the industrial revolution changed this. The culture of mass production and commoditization of labour turned the craft of type design into a mechanically driven process, separating designing and making, in which different teams were responsible for design, punch cutting, type setting, and more.

It wasn’t until the introduction of personal computers in the late 20th century that digital type design became necessary, and once again reunited designing and making, providing the ability for the independent designer to also produce and distribute their own fonts. (Carter)

Despite advancements in type design technology, the largest font projects are still created by corporations and require significant amounts of manpower to maintain. For example, Google’s Noto Fonts collection, which was started in 2012, supports over 1,000 languages, contains over 64,000 characters, and is being added to and updated regularly on Github.

After working on my first font, I started to understand why the design process was so time-consuming. Although there are plugins and scripts available within existing type design programs, tasks such as creating different font weights and adding glyphs in other languages remains a largely manual process. Depending on the size of the family and the complexity of the project, creating a font can range from a few weeks to several years. So, in Thesis 1, I explored how emerging technology could be used to both assist the type design process, and develop new typographical concepts.

Project This project can be organized into a few components. First, I will learn how to make variable fonts and conduct machine learning experiments with glyph style transfer to make the font. There is the audio classification portion, which uses the Spotify API and natural language processing to map font characteristics with audio qualities. Then, I will spend time on web development to host the font for people to interact with. Finally, the presentation aspect includes GradEx setup, recording demos, and sharing progress and documentation.

Impact I’m mostly making this for myself, out of my own interests and areas I’m curious about. During this semester, I plan to share posts and updates online about what I am working on, as outlined in the schedule. Through this, I hope to learn more about which communities find value in my work, and receive feedback on areas to continue exploring.

I am not targeting anyone in particular, but if I were to guess the groups of people that might be interested, I think it could be people who enjoy typography or are interested in type design, but have not really invested the time into learning to create fonts. Maybe also, people who make music videos, people who watch music videos, ML researchers who are curious about applications of their work, prospective Digital Futures students.

Questions Is machine learning overhyped? How do I talk about my project without feeling like it’s hopping on a trend? Is my project considered art? Will I have time to read books? How much do I care about extending the font to support other languages?

---

Adam: When you are communicating feelings with fonts, what are the limits to readability? How can you detect the feelings of a song? Are those feelings the same for everyone? Are they the same for each person every time?

Me: I probably need to change the thesis question to reflect this. I don’t really care about other people’s feelings, only my own. Although fonts traditionally serve a utilitarian purpose in which people need to use them to read things, I take a somewhat artistic approach to fonts with this project – how people react to the visual qualities of characters is more important to me than being able to read the words themselves. (This doesn’t mean the lyrics will be completely unreadable.) Maybe this – “How can I use fonts to share my experience of listening to a song in real-time?”

Adam: Is machine learning overhyped or not hyped enough? Many people envision a future where everyone works with an Al agent or many.

Me: Good point, AI seems like an overused buzzword now, but in the future, I think it will be normalized and commonplace.

Adam: Definitely make time to read books. And make sure you have at least one book you are reading for pleasure to let your mind relax and be happy, that is really important if you want to be healthy and creative.

Me: Thanks, I was feeling like reading is something I want to do, but tend to forego when my mind becomes consumed with tasks. The advice is good to hear. ​

01/15/2023 SheHacks In a team of 4, we built a web app that summarizes news articles from left, right, and centre leaning sources. I learned to use React with styled components. I’ll probably do this for the web app portion of my thesis, but with Next.js instead. We used Cohere API for text summarization. I discovered I can also use their API for sentiment analysis, language detection, text classification, and generation. I’m thinking about using this for my project, but the API was pretty slow when we tried it, so I’m not sure if it will be able to provide real-time output like I want. ​

01/22/2023 UofTHacks I went to this hackathon with the goal of getting help to implement pre-trained models for font style transfer. We tried to clone the repositories for FTransGAN, StyledFontGAN, and DeepVecFont. We ran into many issues with installing packages and dependencies. Often commands can differ based on the OS or hardware, and the authors have not provided a thorough README. Sometimes, the instructions are incomplete or don’t exist at all. So, we tried many different repositories. DeepVecFont was the one with the most success. It took the entire hackathon to finally get a test command working and generating results with sample input.

I learned to connect a Python backend (in this case, a ML model) with a React frontend (Figma plugin) using a Flask server. This would be the architecture of the app.

​ 01/25/2023 Adam: Great work! You've been very productive, and made new friends along the way. Fantastic!

What you describe with the package difficulties is sometimes called dependency hell. What you'll find is that some pieces of software require a specific version of another piece of software to run well. And when you make a big project then you will find that when you are combining two different large projects they will have versions that conflict with each other (some tool needs version 3 and some other tool uses the older version 2 that is not compatible with the new version). What you can start to do is create some patterns that work for you over time, like your discovery of Next.js instead of using Create React App.

Does your output need to be realtime? One of your starting points was music videos, so in this case the pipeline doesn't necessarily need to be fast, just create awesome output with a reasonable workflow.

Maybe your thesis question can simply be about exploring the new possibilities for font design in the age of AI assisted workflows? I think you are right that it is going to be normalized in the near future. Did you find a fiction book to read and make time to read it? (I need to follow my own advice, I'll report back on this to you too)

Me: Thanks for the comments, they have been useful in the past month! Also yes, I have been reading Klara and the Sun by Kazuo Ishiguro. ​

02/04/2023 YouTube I posted a YouTube video about making a font in Figma to see if anyone would be interested in my work, establish the validity of Figma as a tool for typeface design, and build a precursor for my upcoming experiments involving generative models for fonts in Figma plugins.

As a result, 0424, a product manager in the NLP space, reached out to me to learn more about my work. I was introduced to 0425 and 0426, who are working on style transfer for websites, and we may collaborate on a multilingual font style transfer project in the coming months. 0427, a student at Tufts, is adding my video to a design newsletter he runs at school. And my friend 0385 told me she initially started watching my video out of support, but ended up becoming interested and learned a lot of useful workflow tips. People encouraged me to keep sharing my work and I feel more motivated.

What I’m up to now I'm building a Figma plugin that uses a pre-trained ML model, DeepVecFont, as the backend, and learning full stack development using primarily React, Flask, and Python.

It's hard to find a balance between working on my own, and talking to people about my work. I also feel like I haven't had time to focus on code recently, which is what I want to do more of. Although my initial thesis question was about fonts reacting to music data, I'm now thinking that my work is more accurately described as an exploration of creative applications of data science in typeface design, which will include music stuff in the future, but that's not really what I'm building at the moment.

​ 02/13/2023 Notes from my conversation with one of the authors of DeepVecFont – Yizhi, and my friend James.

On font generation projects Most of the font generation work he did in China was funded by a Chinese design company. Part of the motivation was the time consuming process. For example, in designing the custom font for the Beijing Olympics, it took the designer about three years. Contrary to what I had thought from reading 10 or so papers, ML for font design is quite a niche area (10 is really not a lot). So he is researching other areas as well.

On differences between use of ML in English and Chinese fonts Chinese fonts are more labour intensive because there are more characters, so the main use case right now is to save time. With English, there are less characters, so the use case is more about creativity, like using image generation models such as DALL-E or Stable Diffusion to combine with fonts, for producing new ideas. In different languages, the challenges are different. For Arabic fonts, it is also different because it is more like a flowing script. There are different stages, think about historical evolution of font design across languages. And culturally, why was printing slow to spread in Asia despite early invention.

Using data outside of existing fonts Experimenting with popular image generation models like DALL-E, SD or language models has the benefit of being pretrained on millions of knowledge already. With Karmann's tree font, carving a font into the root of a tree is a new direction that in academia no one has tried... at least that he knows of. Calligrams are also something interesting to look into.

On collaboration and application As ML researchers, collaboration and working with designers is valuable, to uncover new problems to work on and bring new ideas and perspectives. It's a combination of deep learning methods and ideas from designers. Some ML researchers are more interested in publishing papers, while others really care about application and want to see their work get applied in more real-life products. Yizhi is the latter – “if we can put font design, logo design into real scenarios, that is ideal." I asked if he knows anyone who is currently working on applying his font generation research into real products… and he said, “You?"

Updates to DeepVecFont V2 will come out some time this year and it will be easier to integrate into web apps. There is another Masters student working on it too. No more DiffVG – there are some disadvantages to DiffVG. Need to use as raster image detail cannot be perfect, to fit the vector graphic with image. There will be more support for Chinese vector – originally presented in China and this is what stakeholders were asking for the most. ​

03/01/2023 Midterm crit The human experience is about communication. Usually when we think about communication, we think about talking to each other. But it goes beyond talking. The world is always changing, and language evolves as what we want to communicate changes, and technology changes. For example, we see this in the evolution of Chinese characters, which have origins in agriculture and farming. With the emergence of new technology and less people working in agriculture, these words start to change meaning.

An example of technology is fonts – digital fonts are relatively recent, as they emerged after the Internet in the late 20th century into the 21st century. Fonts are a tool we invented to express our ideas. Like McLuhan says, "We shape our tools, and our tools shape us." We are limited by our tools, and today's tools are data driven.

​ 03/27/2023 Got DeepVecFont to install and run without errors, but didn’t actually get to test the model with my own input. Got help from my friend who realized we can’t do it on Mac. So we installed Linux, and got it working with some test letters. But, now running into some problems with more complicated letters that have a higher number of paths. The process has been really fun though. ​

04/05/2023 Final crit My thesis questions kept changing from the fall 2022 term, to now – How can we build tools for typeface design that empower the individual to shape the future of communication? How do our tools shape us? How is AI changing the future of fonts? How can fonts be used to communicate the feelings of listening to a song in real-time? What does data science and type design look like? How can I use and make different tools to make different fonts?

Over the course of my thesis, I found many examples of generative typography, read a lot of ML research papers, and books about tools, language, and the evolution of technology. I had conversations with people in the field like Arshia, author of Navigating the Latent, Yizhi, a PhD specializing in fonts and ML, and Jordan, founder of Diagram.

Now, what did I actually do in the last month? I installed Linux and tested DeepVecFont by exporting incomplete fonts. I then wanted to combine aspects of the natural world and man made tools, so I tried growing salt crystals on letters with various materials like wires, wood, and glue. I then digitized the letters, and experimented by inputting some of them into the model to see what it would output.

So far, my thesis has taught me a lot on finding people, on technology and art, and on the role of school and figuring out what I enjoy. What's next is working on a demo website to interact with the fonts, refining the fonts with alternate characters and variable widths, working on documentation, and preparing for my installation.

Not the end... More coming soon to GradEx 108. ​

Related Crystal type GradEx 108 ​