When you are going to design a dialog, you should follow several rules to obtain a successful result. Some rules are listed next:
Never make the dialog contents so large that you need scrollbars to manage it. You can always partition the contents into pages of a tabbed dialog or other paged dialog types. This does not exclude lists and editor widgets inside a dialog. These components need scrollbars, but never make a dialog with 100 pushbuttons.
Never make a menu or a toolbar in the dialog. Menus and toolbars belong to the top level program window. The dialog should be as simple as possible and contain only action buttons (for example, OK, Cancel, Apply) that do something with the selected setting of a dialog.
Use existing widgets whenever possible when you design a dialog. Remember that users need to spend time learning a new interface. You should not make their task more difficult. Most likely, they are already familiar with standard graphical user-interface components and know what to expect from them.
Make the interface as simple as possible. Collect related widgets inside a rectangular frame with a title, or use a horizontal or vertical line to indicate what belongs to what. Never use too many frames or lines. Placing every widget in a dialog inside one big frame makes no sense; rather, it just wastes screen real estate.
Avoid using colors to indicate a state or a setting. Using text instead is almost always a better solution (assuming that your target users can read). Colors can have different meanings in different cultures and some people even lack the ability to differentiate between certain colors, such as green and red. Do not exclude those users from your potential user base. Stop is better than red.
If you are interested in more information on how to do it right and how to do it wrong, visit the Interface Hall of Shame on the Web, http://www.iarchitect.com/mshame.htm, and learn from mistakes other developers have already made. You don't have to reinvent others' mistakes.