To switch between formations, or rearrange your characters within a formation, click the Formation button on the bottom of the screen. The game remembers your character order for each formation, so if you later switch back to a formation you've already set up, you'll be ready to go.
You can only use formations that you've leveled up to 1 or higher.
The thread What formation do you use? on the official forum has a bit of useful advice, but it's mostly from people early in the game.
To level up a formation, click the Talent button on the left side of the screen, then Formations on the bottom.
Each formation has a minimum main character level you have to reach before you can enable it.
Level 1 unlocks the formation, allowing you to use it. Additional levels give you nothing but speed.
Fleetfooted is special; it's not a formation, but a speed bonus that adds to every formation (but only 1/20th as much as a formation level).
Each formation provides a bonus to a particular stat. These bonuses are pretty small, and they're fixed numbers rather than percentages, and they don't increase with formation level. So, very early on they can make a difference, but at higher levels you won't even notice them.
Each formation gives you 5 slots to put characters in, numbered from I to V. This is the order your characters will attack in. On each turn, you will normally attack the frontmost enemy character in your column. Column, row, back, and cross attacks are also based on your position.
So, there are many things to take into consideration. You may want to put your best damage-dealers behind tanks (not just guardians; companions like Andre and Lulu can also make effective blockers), line up a glass cannon to one-shot an opposing skill-all enemy, avoid exposing three weak characters in a column to a strong column attacker, etc.
For tough fights, you will have to watch the fight carefully to figure out how to rearrange your team the next time. If your Andre who starts at 80 rage isn't getting hit before his first turn, move him; if a column attack is killing Martina before she even gets to move, put her in a different column; if a damage-dealing monster is hiding behind a guardian, make him hit someone with high counter and line up an assassin to hit him from behind; if your column attack is only hitting one monster, hit a different row; etc.
When selecting a formation, across the top it will say something like "SPD410". That means you have 410 speed in that formation—100 base, plus 300 from getting the formation to level 3, plus 10 from getting Fleetfooted to level 2.
In PvP, whoever has a higher speed goes first. In PvE, speed is irrelevant; you always go first.
Do not level up all of your formations equally.
Sometimes it can be better to go second, if you rely more on physical/magical damage and countering than on skill blasts. You can also use this to get some extra rage on your frontline companions, unless your opponent is relying on tranquilizing.
But usually it's better to go first. Especially if your server has reached the point where nearly everyone you fight has Ying Chen or a main or other companion who starts at 100 rage, tranquilizes the other side so they can't return fire, and does a lot of damage.
Pick one formation that's good against most of your enemies and level that one up as far as possible, and add some Fleetfooted levels to get ahead of the other people who've reached the same formation level as you. Most people seem to pick Ferocity, Evasion, or Fortitude; you can do the same, or try to pick something that makes it easier to counter those. Since formation level costs are exponential, you may want to get another one or two formations reasonably high for countering specific opponents. But you're going to specialize in one, and once you've picked it, you're locked in for a long time.
Keep some other formations slow, that are particularly good for combatting layouts you see a lot of. If there's someone you can't out-speed, maybe you can out-counter him. Dexterity, Agility, and Fortitude are frequently good. (Yes, Fortitude is a good fast formation and also a good slow formation; you'll have to pick which way you want to use it.)