While Federal troops labored to build the dam, Taylor’s Confederate army was not idle. Some rebels continued to harass the outposts around Alexandria, while others destroyed bridges and blocked roads in an attempt to entrap the Union forces. Federal boats already below the rapids were constantly ambushed along the lower Red River as they attempted to supply the army. In fact, the Confederate soldiers were able to cut off all navigation on the river for a while, isolating the Yankees.

On May 13, with all the gunboats now safely below the rapids, Union forces moved out of Alexandria. The Union soldiers left with mixed feelings. They had been beaten in battle, harassed, and almost completely destroyed. They were exhausted. Still, they had accomplished a magnificent feat in building the dam and rescuing the fleet, and some had even made friends among the townsfolk.

But for the local population this was a critical time. Rumors spread that the town would be torched when the army left. Banks ordered a detail of 500 men to be left behind to protect the town from arson. But fires quickly broke out as soon as the main army was out of town. It is unclear who started the fires, as some accounts describe soldiers looting and setting fires, while other accounts note that army guards shot looters. Probably, both Union troops and local looters were involved. One detachment, the 92nd Colored Infantry who also helped build the dams, was known to have fought a fire for many hours, until the building was doomed and the troops were forced to continue their retreat. One Yankee soldier described the scene:

Cows ran bellowing through the streets. Chickens flew out from yards and fell in the streets with their feathers scorching them.... Crowds of people, men, women, children and soldiers, were running with all they could carry, when the heat would become unbearable and dropping all, they would flee for their lives, leaving everything but their bodies to burn. Over the levee the sights and sounds were harrowing. Thousands of people, mostly women, children and old men, were wringing their hands as they stood by the little piles of what was left of all their worldly possessions (Van Alstyne 1910:320-321).

As the expedition retreated south down the Red River, Confederate cavalry did what it could to badger the Union forces at every opportunity. However, no matter how courageously the men fought, the rebel army was too small to seriously oppose the retreat of the entire expedition. At Mansura, Louisiana, Taylor attempted to stand against the Federals, but after a four-hour artillery duel, he had to withdraw.

Though the campaign seemed about at an end, Banks found that he had to call on the services of Lieutenant Colonel Bailey once more. At the Atchafalaya River, Bailey directed the construction of a bridge which he promptly fashioned out of transport vessels. Around 24 transports were placed across the river and held together with large timbers. Then, long planks were laid across the prows of the transports to form a temporary bridge. Banks’s army was able to cross the river quickly and safely and continue the retreat south to Baton Rouge. By that time, every soldier knew and appreciated the frontier engineer from Wisconsin.