If it doesn't do it without the ladder up there, but it does when you strap on the ladder, I think you've found your culprit for sure. Maybe you could try swapping the ladder end for end - That should make it better . . . . or worse, but at least you'd know for sure it's the ladder causing the problem
I had a little Mazda wagon that had roof crossbars which I very seldom used and just the front crossbar made an unusual noise that I wasn't at all happy with. The very next year on the same model, Mazda addressed the problem by adding little rubber 'spoilers' on the underside of the crossbar. When I complained to my dealer about the noise, they ordered and installed the newer model crossbar for me and while it didn't eliminate the problem entirely, it did lessen it to the point that nobody but me noticed it any longer
Pretty much anything on the roof at freeway speeds is going to make some noise - Some of them more objectionable than others. All the more so in a vehicle with little or no insulation in the headliner , , , , or in your case, no headliner at all
Don