Cartagena, Colombia (CNN) — Twelve Secret Service agents sent to Colombia ahead of President Barack Obama were relieved of duty and returned home amid allegations of misconduct that involved prostitution, the man who tipped news agencies to the investigation told CNN Saturday.

The incident in Cartagena — one of two security issues that occurred Friday — overshadowed the start of the sixth Summit of the Americas, where the president was to focus on trade, energy and regional security with 33 of the hemisphere’s 35 leaders.

“One of the agents did not pay one of the prostitutes, and she complained to the police,” said Ronald Kessler, a former Washington Post reporter and author of “In the President’s Secret Service: Behind the Scenes With Agents in the Line of Fire and the Presidents They Protect.”

The Washington Post, which was the first to report the story, said it was alerted to the investigation by Kessler.

Prior to the president’s arrival, an undisclosed number of Secret Service agents were relieved of duty and replaced, said Edwin Donovan, an agency spokesman.

“There have been allegations of misconduct made against the Secret Service in Cartagena, Colombia, prior to the president’s trip,” Donovan said in a statement.

“Because of this, those personnel are being relieved of their assignments, returned to their place of duty, and are being replaced by other Secret Service personnel. The Secret Service takes all allegations of misconduct seriously.”

Donovan declined to identify the nature of the alleged misconduct, saying only the mater was being turned over to the agency’s internal affairs.

But Jon Adler, president of the Federal Law Enforcement Officers Association, told The Washington Post that the accusations relate to at least one agent having involvement with prostitutes in Cartagena.

CNN could not immediately confirm the claim.

Calling it “clearly the biggest scandal in Secret Service history,” Kessler said the 12 agents are accused of involvement in the incident “in one degree or another,” from allegedly interfering in the investigation to participating in other alleged misconduct.

Kessler did not identify to CNN who provided him with details of the investigation.

The president arrived in the Colombian coastal resort city Friday, a visit that will mark the most time a U.S. president has spent in that country, where security concerns had limited previous presidential trips.

Amid the reports that Secret Service agents were being replaced, two small blasts occurred nearly back-to-back in Cartagena.

The explosions, one near a bus station and another near a shopping mall, occurred well away from where the world leaders were gathering for the start of the summit, said Alberto Cantihho Toncell, a spokesman for the Colombia National Police.

There were no casualties, and only minor damage was reported, Toncell said.

The explosions came on the heels of a similar one earlier in the day near the U.S. Embassy in the capital city of Bogota, authorities said.

The blasts were a reminder of the violence that has gripped Colombia as it battled powerful cocaine drug cartels. Violence has significantly fallen off in recent years as the Bogota government, aided by U.S. extradition efforts, has successfully picked apart the cartels.