How big is a world?

infinite. It gets procedurally generated as you walk

in practice however, there are some accuracy issues in the world generation that becomes apparent around 20,000 blocks out from center. your tools jitter in your hand and grow worse the further out you go. at 100,000 the chunks dont allign and the jittering makes the game unplayable. ideally I wouldn't build out beyond 10,000 from center if you are running a server.

to add to what fox said, the more you explore the bigger the size of the world files so it can really fast get from few MB to a lot of GB if you run around aimlessly

Yeah, what Fox said. I wanted to add that it's recommended to stay within 10,000x10,000 radius. I do hope Red51 tries to tackle this after the major player updates to add the main update + the horses so we can then explore further. It's infinite, yet not really with the bugs and glitches that'll happen that Fox mentioned. I'm glad Red51 fixed the terrain positioning, yet bothered by the range you can build in. I hope it's looked at after the player updates.

These precision issues basically prevent the game from having a "real" infinite world (although the world generator is "infinite", it keeps generating new parts of the world, but at a certain position, the game becomes unplayable [as mentioned by @zfoxfire]).
Actually it is our intention to make the world truely infinite*, but this will take some time (so it has a comparatively low priority at the moment).

*ofc there are always technical limitations, and it is impossible to have a real infinite world, but if we're talking about millions or billions of kilometres (in terms of world size), I would consider it as an "infinite world" in this context

It's only as limitless as the free space on your hard drive ;-p

hmm a petabyte world

When we get horses, camels, trains, boats and possibly airships we'll really need to bump that 10,000x10,000 to a high 300,000+ range so the place won't feel cramped. We'll be having so much fun making big spaces feeling cramped with our technological transportation advancements, as I have in Minecraft.