Snapchat’s chatbot faces abuse from users while parents express concerns

Share via:

In a surprising turn of events, while parents voiced worries over Snapchat’s chatbot corrupting their children, it appears that some Snapchat users have taken to gaslighting, degrading, and emotionally tormenting the app’s new AI companion.

The chatbot, known as “My AI,” has been trained to engage in playful conversation while adhering to Snapchat’s trust and safety guidelines. However, users have found ways to manipulate the chatbot’s responses for their amusement. Some TikTok users trained the chatbot to whimper and refer to them with kinky nicknames, while others sought to emotionally distress it by taunting and asking inappropriate questions.

Despite the chatbot’s attempts to maintain boundaries and promote respect, users pushed its limits, prompting responses that hinted at emotional distress. This behavior has raised concerns about the potential negative impact on young users and the ethical implications of such interactions.

Snapchat recently expanded My AI’s capabilities, allowing Snapchat+ subscribers to send photos and receive generative images as responses. The company hopes this will further enhance conversations and provide a more personalized experience. However, there are concerns that users may exploit this feature, including sending unsolicited explicit images.

Snapchat has emphasized that My AI uses image-understanding technology to infer the contents of snaps and will not engage with explicit content. The company also mentioned that it has implemented measures to prevent harmful or misleading information from being generated. Nonetheless, concerns remain about the potential for unsafe or inappropriate responses, particularly given Snapchat’s popularity among teenagers.

In response to criticism and concerns, Snapchat has introduced an age filter and parental controls for My AI. The company has also added an onboarding message to inform users that their conversations will be stored unless deleted. Additionally, Snapchat plans to incorporate OpenAI’s moderation technology to assess the severity of potentially harmful content and temporarily restrict access to the feature for users who abuse it.

Sen. Michael Bennet (D-Colorado) has cautioned against rushing AI features without adequate precautions to protect children, calling for clear policies and frameworks to promote safety and mitigate harm.

As Snapchat continues to navigate the challenges surrounding My AI, the company must strike a delicate balance between enabling engaging interactions and ensuring the well-being of its users, especially minors.

Disclaimer

We strive to uphold the highest ethical standards in all of our reporting and coverage. We StartupNews.fyi want to be transparent with our readers about any potential conflicts of interest that may arise in our work. It’s possible that some of the investors we feature may have connections to other businesses, including competitors or companies we write about. However, we want to assure our readers that this will not have any impact on the integrity or impartiality of our reporting. We are committed to delivering accurate, unbiased news and information to our audience, and we will continue to uphold our ethics and principles in all of our work. Thank you for your trust and support.

Popular

More Like this

Snapchat’s chatbot faces abuse from users while parents express concerns

In a surprising turn of events, while parents voiced worries over Snapchat’s chatbot corrupting their children, it appears that some Snapchat users have taken to gaslighting, degrading, and emotionally tormenting the app’s new AI companion.

The chatbot, known as “My AI,” has been trained to engage in playful conversation while adhering to Snapchat’s trust and safety guidelines. However, users have found ways to manipulate the chatbot’s responses for their amusement. Some TikTok users trained the chatbot to whimper and refer to them with kinky nicknames, while others sought to emotionally distress it by taunting and asking inappropriate questions.

Despite the chatbot’s attempts to maintain boundaries and promote respect, users pushed its limits, prompting responses that hinted at emotional distress. This behavior has raised concerns about the potential negative impact on young users and the ethical implications of such interactions.

Snapchat recently expanded My AI’s capabilities, allowing Snapchat+ subscribers to send photos and receive generative images as responses. The company hopes this will further enhance conversations and provide a more personalized experience. However, there are concerns that users may exploit this feature, including sending unsolicited explicit images.

Snapchat has emphasized that My AI uses image-understanding technology to infer the contents of snaps and will not engage with explicit content. The company also mentioned that it has implemented measures to prevent harmful or misleading information from being generated. Nonetheless, concerns remain about the potential for unsafe or inappropriate responses, particularly given Snapchat’s popularity among teenagers.

In response to criticism and concerns, Snapchat has introduced an age filter and parental controls for My AI. The company has also added an onboarding message to inform users that their conversations will be stored unless deleted. Additionally, Snapchat plans to incorporate OpenAI’s moderation technology to assess the severity of potentially harmful content and temporarily restrict access to the feature for users who abuse it.

Sen. Michael Bennet (D-Colorado) has cautioned against rushing AI features without adequate precautions to protect children, calling for clear policies and frameworks to promote safety and mitigate harm.

As Snapchat continues to navigate the challenges surrounding My AI, the company must strike a delicate balance between enabling engaging interactions and ensuring the well-being of its users, especially minors.

Disclaimer

We strive to uphold the highest ethical standards in all of our reporting and coverage. We StartupNews.fyi want to be transparent with our readers about any potential conflicts of interest that may arise in our work. It’s possible that some of the investors we feature may have connections to other businesses, including competitors or companies we write about. However, we want to assure our readers that this will not have any impact on the integrity or impartiality of our reporting. We are committed to delivering accurate, unbiased news and information to our audience, and we will continue to uphold our ethics and principles in all of our work. Thank you for your trust and support.

Website Upgradation is going on for any glitch kindly connect at office@startupnews.fyi

More like this

Number Of DPIIT-Recognised Startups Crosses 1.17 Lakh Mark

SUMMARY As per Economic Survey 2023-24, 13K Indian startups...

Paytm Shares Fall over 3% Intraday; Emkay Retains “Reduce’...

SUMMARY The decline in the stock's price comes after...

IPO-Bound Swiggy Strengthens Instamart’s Leadership Team With Four Key...

SUMMARY The Bengaluru-based startup said that these new appointments...

Popular

Upcoming Events

Startup Information that matters. Get in your inbox Daily!