Custom coins are no different. When we mint coins, a blank metal disc is dropped into a collar. A collar is a ring that keeps the metal from squirting out the sides with it is struck with the tons of force needed to produce a die struck coin. The inside edge of the collar can be smooth or have ridges.
When the blank is squeezed between 2 dies, the metal is forced up in the dies (to create the image) and out into this ring, making the edge of the coin take on the shape cut into the collar.
Think about it similar to a waffle press - the batter takes on the shape of the press - whether it's the "standard" waffle you get at the Waffle House, or one shaped like a death star or Mickey Mouse. Sometimes you put too much batter in and it oozes out the sides. A collar contains the metal and also gives it an edge - hence the reeded or plain.
Making it even more complicated - there is something called an "open reed", which has reeds 75% of the way around the coin and then a smooth section for the remaining 25%. This open reed allows an area when a coin can be edge numbered (0001, 0002, 0003 etc) or have words like .999 FS (.999 fine silver) stamped on it.
Whichever you choose, this is a no charge option. If you want a plain edge or a reeded edge, there is no additional cost.
Talk to your salesperson about what is right for you.
Here are some additional resources: