I have the same setup on my camper. The 20 watt panel is just enough to keep the batteries up while the camper isn't being used. I added an additional 80 watts worth of panels which give me an extra 6 amps for the batteries in full sun. That's a nice bump and does bring the batteries up to a full charge by the end of the day as long as there's not a lot of power usage throughout the day. As far as a perfect size for you, that depends on how much the camper is being used and what accessories as well. I am thinking the optimal size for me would be an extra 200 watts (~15 amps). probably 100W x2 panels. Since the built-in charge controller will NOT handle that much additional power, it will either need to be replaced or an additional one added. I took the easy route and added a polarized connector to one of my batteries so that when I get to my destination I can plug in a charge controller with my 80w panel.
Just a couple of tips:
Mount your charge controller close to the batteries and use good quality fairly heavy copper wire. The shorter the wire run and larger wire the better as this will cause less of a voltage drop. That being said, you don't need to go overly large on the wire. I used 10ga which is plenty.
This also allows for longer wire runs with a smaller gauge wire out to your panels. Since your panels will produce a voltage that is normally many volts higher than your battery voltage, a little bit of a drop is insignificant.
When I get another panel, I will be using 2 separate 30ft. wires from my charge controller out to the panels. This will allow me to move the panels around separately rather than having them tied together electrically at the panel with minimum movement. It's been my experience that the panels need to be moved around quite often if you're in a timbered area in order to keep getting full sun.
One last note. Although pwm charge controllers are cheaper and they do work well, I would recommend a mppt controller if you can afford it, do a lot of camping, rely heavily on your panels and are in an area where the sun may be limited. I use a pwm on my camper and mppt on my house and love it. I will be upgrading to mppt on the camper soon. I would recommend doing your research before selecting a controller and panels to see what will be right for you. You may want to calculate the amount of power being used by the camper and accessories and then determine how much power it takes to bring the batteries back to full charge. Unless your camping in the desert, you can probably only count on 5-6 hours a day of full sun on your panels.