Watched a user try out our product for the first and wanted to scream at him

A few months ago, I was building my SaaS when I decided I want to do some early user testing (always a good idea). I liked the design of my page very much and thought it was simple and easy to use. So, I asked a friend to sit next to me and do a few tasks that I had planned to test. But he somehow was struggling so hard, he couldn't do the easiest things. I changed a few things, but I tried with other test users and some of them had the same problem. And no it's not because of my bad ui design, but rather b