I don't agree with this. Perhaps you are saying this because you didn't have to take algebra because it was too easy for you. However, I think it is reasonable to suppose that if someone has questions about the difficulty of a following course then the previous course was not easy for them.
After teaching both calculus and algebra it is my opinion that the biggest problem students have with calculus is that they don't have a good grounding in algebra. There are alot of basic calculations you have to do with powers of variables and with polynomials that if you don't have a firm foundation with them then you will have great difficulty with them in calculus.
I really must say I don't agree with what you said about memorization either. Again some students find calculus easy and they don't have to do much work, such as memorization, to do well in it. But that is not true for many students.
Prior to taking calculus the best single advice I can give is that you are strong in doing calculations with powers and polynomials. How do you add them. What are like terms. How do you multiply polynomials. How do you divide them. When adding fractions of numbers or of polynomials how do you find the common denominator.
Also extremely important is knowing how to do compositions of functions, i.e, functions of functions.
Bob Clark