Vtubing app with Decentraland Avatars. This proposal is now in status: REJECTED. Voting Results: * Yes 41% 4,623,770 VP (124 votes) * No 59% 6,504,612 VP (90 votes)

Thank you for this point Morph. [quote="Morph, post:12, topic:17340"] If the proposal was changed to provide an open sourced library or website that accomplishes the above and exports a useable .vrm file from current avatar, I would gladly vote yes as I think this is incredibly valuable to the community. [/quote] Decentraland is open source and I prefer that as many development grants as possible be open source too. The recent [2022 year in review blog post](https://decentraland.org/blog/announcements/decentraland-2022-recap/) describes Decentraland as a public good. Fewer walled gardens funded by the DAO. More extendibility instead.

This is a sick idea that we were thinking of as well. We work with a lot of DJs in New York, Berlin, and Thailand. Would love to partner with you on this.

I think this proposal is not being well understood, and it seems to me a MUST HAVE. This app would provide more value than many games, how? Linking your virtual identity to your decentraland avatar. I have been one of those who has asked swiss how to do it, I have also tried the application that hkprivakos suggests (paid, and great information in Japanese). I have messed around a weekend and I have not been able to, so I have moved on to something else. I am not a 3d designer and that is a big problem to even understand what I need to get started. Then there are many free programs where you waste hours trying for nothing. Now, as I understand it, what is the potential of this app? I give you an example. Call with friends, I appear with the decentraland avatar. Wow how do you do that? Well, with a decentraland avatar and this app. Oh, I want to try it. Imagine if a famous youtuber uses a decentraland avatar, do you think it won't attract users? Today there may be little interest, it is very difficult to find an audience creating content about decentraland... but helping users to link their decentraland identity with their real identity... isn't that what it's about? For now I would use it, I'm sure Kevin would too, but it's not just a tool for streamers, it's a tool to chat in a discord with your avatar, in a meet... I see it as super powerful. And I see it as a tool that will do free marketing for decentraland as soon as it is used. Regarding the wallpapers, I would suggest two options, such as twitter, having your image downloaded from the internet without validation, and an image with a market of images perhaps linked to wearables or independently but within the decentraland ecosystem. If you have any collection from the Spanish museum, you belong to that community... you could download the jpg and have a mark that identifies if you have an nft or it is a simple image to give it the maximum potential. I honestly think it's a different proposition that adds value and it's not a crazy price for a group of people and the value it can bring. There may be few people asking about this feature, but I don't doubt that people who don't even ask right now will use it, and maybe most of them dont know about it or think it is so hard as reason to dont ask. again, isn't that what it's about? Nobody could program, or design 3d, but everybody could share a little video with its decentrand avatar.

Can this be put up for Bid and Tender, and have Tony have a team to execute this? I think there is huge value but its a big task for one person to think about all directions. Can we please give a HEALTHY amount of funding for the brave warriors willing to build this out? THANK YOU TONY!!!! <3

Hi Tony, thank you for all the info! I love what you're building and can see it's had a lot of work put in - I have some more thoughts and curious to hear yours. Glad to see the greenscreen background - I'm not sure having a background market would necessarily be an immediate need, but sounds like it could be a cool way to get avatar backgrounds and connect artists with users (I've mostly used Fiver or open licensed stuff so it is definitely a viable use case), perhaps this could connect to your wallet and you could use any NFT image you already own? This way we are leveraging all NFT marketplaces and the already existing economy so we don't need to spin up a net new 'image' marketplace. There is still great service layer potential here as I believe the 52 blend shapes in ARKit should somewhat sync up with the current open standard of VR facial expressions - (May require a re-mapping of shapekey names but that should be arbitrary). I'm curious where your application will store and access the shapekeys, and how users will contribute them? The DCL marketplace already accomplishes a majority of the heavy lifting in terms of connecting all our systems for .glb files, **ideally we can extend that to enable wearables with shape keys instead of a net new facial avatar system** as well as an API that any user or application can use a metamask signature to download the user's body/facial-tracking compatible avatar - I don't think these two ideas of the app layer and service layer conflict, and could even build off of each other to be more valuable, but we should ensure the work is interoperable for VR clients as well as building on the current .glb network to not duplicate efforts. I think it would make a lot of sense to take the learnings from this project, and use it to build the underlying system for creators and users before moving forward with application layers. I really admire what you've built with OpenCV, I'm more versed in computer vision than I am 3d modelling so I appreciate how much effort has gone into this - I think it is the next natural step of evolution for our metaverse wearable marketplace and avatars, and am thrilled to see more contributions in this space. **Re: Web3** The web3 argument you make is accurate but there is always boundaries of compromise and a scale of decentralization due to the needs of an interoperable metaverse, we have to allow applications like DG mobile poker to have access to all assets of all the avatars playing for example. I think one reason we cannot genuinely tokenize everything individually is because many of it is already tokenized, long term, how will other users from other metaverses bring their .vrm to DCL applications like this and our VR apps, or more importantly, how will DCL avatars travel to alternate metaverses if those other metaverses took the same approach? What if a user builds an avatar from wearables from multiple metaverses? I don't think I can 're-mint' my own avatar every time I want to try a new VR metaverse, it would not only be a logistical and legal nightmare to not tread on any creators copyrights, but also a massive UX issue that would likely have ended my exploration before it begun - if we go this route I'm not sure I see a massive difference over something like Meta, what good are the NFTs if the entire system is centralized to control and restrict the access to them?

[quote="HPrivakos, post:20, topic:17340"] I couldn’t find any mention of open source in this proposal, sorry. About the blend file, it can be generated in a few minutes with Blender and an open source plugin, not a huge achievement or proof of anything. [/quote] The blender file is a proof of transparency regarding the development process, precisely by seeing the limitations and requirements of using the blender file, that's how we decided to create the app. - The avatar had an adaptation process, the default mesh is not compatible, we had to bake and create new materials using vertex painting. - We created the 52 blendshapes manually to make it compatible with ARkit. - We precisely used the OSC protocol to be compatible with the blender interface (OSC was the only open source tool we used in the process). - The file has a camera rig to be able to control the focal length, focal point and tracking only using emptys in a simple way. - Through collections we can change the background with a click. We have a setup in the environment node that allows us to have a green background without affecting the lighting of the scene. Can you tell me which free plugin can do this in 5 minutes? I don't understand your position with the last argument, seriously, basically making it web 3 is the main advantage, not just talking about the app, but in general, the land, the dcl names, the wearables, could work without web 3, in fact decentraland partly does, at 8:13 of the AMA https://www.youtube.com/watch?v=kD3X4wm3fiE&t=1030s they explain how Decentraland works in a technical sense (the centralized part and the web part 3) also it is proposed to use the SDK as an "open standard" under your argument, someone could make a fork and have a version of decentraland with all the wearables for example, even so doing that would lose all meaning, I mean? If you are in some contact with the community of vrm avatar creators, it is common for a user to download your model illegally, precisely tokenizing them is an excellent option for that 

[quote="TonyDarko, post:19, topic:17340"] We said it from the beginning, everything will be open source [/quote] I couldn't find any mention of open source in this proposal, sorry. About the blend file, it can be generated in a few minutes with Blender and an open source plugin, not a huge achievement or proof of anything. [quote="TonyDarko, post:19, topic:17340"] it will be compatible with any vrm that can be added within the nfts of the marketplace [/quote] I find it extremely counter productive to limit on purpose the functionalities of an application just to make it "web3". There is absolutely no good reason to voluntarily put a limit to which VRM can be loaded and to use a NFT for that. But if it's open source, I don't really care as someone will be able to fork it and remove that limitation.

[quote="HPrivakos, post:17, topic:17340, full:true"] Will everything be fully open source or do you plan to keep some parts proprietary? And will it support solely DCL avatars or all VRMs? [/quote] We said it from the beginning, everything will be open source, I even left the blend file of my first tests in eevee in real time for the community, as well as in the other proposals we will be transparent with the whole process. On the other hand it will be compatible with any vrm that can be added within the nfts of the marketplace, again, we put that in the documentation as an extra point.

You are wrong, let me explain how Webcam Motion Capture works: - Webcam Motion Capture sends signals under the VMC protocol to Vseeface, which means you need two applications. In addition, you need OBS to be able to have the virtual camera, which means you need three software programs to perform the same function. Also, Webcam Motion Capture is not free; you can try it for free, but you have to pay to use the full tools of the app. And what are you going to say? That it is cheaper to pay for the app than the grant? What sense does it make to pay for something if you still need more steps and it will be more complicated to use than our app, which will be completely FREE and also have all of these same attributes in a single executable? If you think about it, this will bring more users to decentraland. Also we think is better to use the OSC protocol over the VMC protocol could be because OSC is more widely supported and has a longer history of use, OSC is a protocol for communication between computers and other devices that was developed in the late 1990s and has since become a standard in the industry. It is supported by a wide range of software and hardware platforms, making it easy to integrate into a variety of systems (for example decentraland, check this repository https://github.com/decentraland-scenes/osc-relay) In contrast, the VMC protocol is a relatively newer and less widely used protocol. It may be more difficult to find software or hardware that supports it, which could limit its usefulness and flexibility, OSC has a more robust feature set and is more flexible, allowing for the transmission of a wider range of data types and the use of custom data formats. OSC may be the more reliable and flexible choice for communication between devices and systems, making it a better option than the VMC protocol.

Will everything be fully open source or do you plan to keep some parts proprietary? And will it support solely DCL avatars or all VRMs?

[quote="TonyDarko, post:15, topic:17340"] of course vseeface allows you a facetracking but only that, we developed a hand tracking as well, yo can see our development status above. [/quote] It does. And with a webcam, no need for DSLR. https://www.youtube.com/watch?v=OHI1ZnIP558

@Sannin Hi, thanks for commenting, that's right, we decided to separate the proposal into 3 at Esteban's request, (we had one) but the vtubing app is the mother project, the total budget includes all aspects of development, despite not having received the grant we have already started working on the app, this is the current status, we need to polish and work on many aspects, in a few hours we will share more about the status of the other 2 proposals: https://twitter.com/TonyDarko_eth/status/1607550656079958016?s=20&t=KzEYJcVPyodUV93KEEb6dA @Morph Hello, although few you are one of the people I have seen in the community with an interest in vtubing, this app is for that, of course vseeface allows you a facetracking but only that, we developed a hand tracking as well, yo can see our development status above. We have decided to add in the development a tool that allows to export directly from blender vrm files, now talking about your second comment, the base rigg of dcl, vrm and even mixamo are practically identical, but they have different names, the fbx of the dcl documentation has the eyes and mouth in png and not mesh, it is necessary to add a bone ROOT to the dcl rigg to not have compatibility problems and that requires further development, for that reason it is better the first idea of having a native converter when developing wearables, as you said so we are going to work on it

First of all happy holidays to all, sorry for not answering sooner, I was with the family these last two days. @HPrivakos I don't understand your first argument, basically without something already existing you can't tokenize anything, in fact that's the purpose, on the other hand in the documentation we wrote other points in favor to be able to use inside the app which are: - **Being able to tokenize vrm avatars**: this could be a very important point for the community of creators and also for the dao to increase transactions. - **Virtual camera/calls inside the app:** Most of the apps in the current market do not generate a virtual camera in a simple way or they need a software like OBS to do it, we are developing the simplest way to do it with the best possible result. I agree with the usefulness of converting mesh from wearables to vrm, we can include that development within the proposal.

We ideally need open source tools and frameworks to be built over applications. I don't think we can realistically expect Decentraland to rebuild every tool it needs when open source and interoperable versions are available. We will consume a large amount of tech debt if we focus on building out applications over generic and open source frameworks, we need to power creators in Decentraland to build and explore the metaverse, not a locked down ecosystem where experiences are token-gated - at least I think that's the vision of the 'free and open metaverse' over of something like Meta. "The Avatar pipeline" is a known unsolved problem in the metaverse, no application let's you build a VTuber avatar from a marketplace of open sourced and interoperable wearables/assets that the community or anyone can create and contribute to (and has it's own economy). The closest most will get is shelling out 2-5k USD for a custom built one on Fiver. This is however, a problem Decentraland is uniquely positioned to solve, as we have 90% of the pipeline created and verified by the blockchain already - Our avatar wearables are even based on the same skeleton as the open source default VR avatar formats (.vrm), we are only missing a way to map the facial expressions and export the file extension. I implore you to consider re-proposing this as building out an npm package for the DCL repo that developers can use to build VR compatible exports of custom wearables by providing shape keys/mapping, it will open up interop for the entire community and let people experience the metaverse as it should be. This library could then be added to the DCL-core wearables pipeline, so creators can upload facial expressions alongside their .glb file for 100% end to end UI-automated interoperability with applications like VRChat - I truly believe this is the future of DCL wearables and avatar creation, but in my limited skillset it's taking myself a long time to get there - I hope you consider this request.  In my research so far I believe a proof of concept would follow these steps: - Download current avatar .glbs from current DCL endpoint (verified by metamask signature - or just use a locally hardcoded return value for demo). - Provide a JSON interface for mapping shape key files to wearable slots (eyes/mouth wearable slots doing basic movement would be enough) - Export the configuration as a .vrm file The full pipeline would add more in-depth shape keys and a strategy to merge the shape key implementation and .vrm export with the current DCL wearables pipeline, a feature that I believe would increase the metaverse value prop by enabling DCL VR avatars for both external use and for when we have our own VR client. Apologies for the word dump, I hope this was helpful.

Hi there, I love the concept as I currently VTube as my DCL avatar by exporting my avatar into .vrm format and using VSeeFace and other open source clients. While I think this would be useful, I would be much more interested in something that takes our current avatar endpoint (.glb files) and converts them to a .vrm file to be used in many current opensource facial tracking and VR clients. It seems your team has the skills and capabilities to achieve this and fulfill the full mapping of mouth/eye/eyelid shape keys to the default avatar faces - it would be great if a module was added so that wearable creators or the community can provide the shape-keys alongside their .glb files - as there is a large number of both default and custom eyes/mouths/faces available (some face replacements are masks/helmets so need a way for community artists to upload and also map shapekeys that they create for their .glbs). The idea of NFT backgrounds sounds limiting and frustrating, VSeeFace enables a transparent background to use any image, if users wish to purchase a background NFT and use that image they can do so, else limiting creative freedom and interoperability. If the proposal was changed to provide an open sourced library or website that accomplishes the above and exports a useable .vrm file from current avatar, I would gladly vote yes as I think this is incredibly valuable to the community. A proof of concept would be required due to the technical complexity, but given this requirement I believe a higher grant amount would be understandable. You can read some of the communities current thoughts on this topic here, a proposal that passed overwhelmingly: https://forum.decentraland.org/t/dao-qaudhgm-enable-vrm-support-for-decentraland-models/14208

Voting no, as you are asking for 3 separate grants (60k USD total) and imo you should show proof of concept or get interest in your work from community first.

I think Tony is really talented, I'm in love with his work. However, after reading the whole discussion, I don't find this project very appealing since the market is very small and there are already free tools to do something similar.

Hello @HPrivakos, I disagree, I think yes makes sense to make web3 applications of thing that already exists because that is the meaning of toquenize things… people could be creative with backgrounds same way like NFT art… Think… money already exist but Bitcoin makes sense, jpg exist but NFT art makes sense, virtual worlds exists but we have here Decentraland… The proposal could open a news way to create and interact in the marketplace what I think is very positive. Thanks for the open discussion

I really don't see the utility in making a "NFT/Web3" version of something that already exist. [quote="TonyDarko, post:4, topic:17340"] Being able to tokenize the backgrounds [/quote] or making it worse that something that already exist by trying to tokenize everything? Just let people use any jpg they want as background for free. [quote="SWISS, post:5, topic:17340"] I would much rather have a .vrm avatar export tool that delivers an a vrm file from any wearables combination on demand. [/quote] That can probably be done for a lower tier grant. Thanks for your reply Swiss

On the other hand, as an addition to the app, there is the NFTs integration with what I already explained, just as you say, polygonal mind released an app for its avatars, if it were easier to have an avatar in any other app, why get your own? The idea is to lay the foundations for further development within that app

Your point seems interesting to me and I partially agree with it, I think that having a basic knowledge of 3D you can achieve having a vrm avatar in a relatively easy way, but for someone who has zero knowledge it is a problem, that is, they know the steps to follow But how could you do it if you don't understand what a shapekey is or the difference between a glb and a vrm? That limits many people and I think that when we have a certain level we take terms or points that other people are unaware of. The purpose of this proposal is to have the best result as simple as possible, just turn on your camera and open the app, and that's it, the more accessible it will be, the more widely used technology will be, with this I don't want to say that everyone should learn 3d, precisely the That a person without even knowing what is a shapekey but he can use an avatar is a great advance, I don't understand how microwaves affect water molecules, I just press a button and that's it, hot water, let me understand?

Hey all. The most complex thing to overcome is actually the creation of a .vrm avatar counter part. Everything else is widely available on the web and a few DCL community members have used avatars for that purpose. Depending on how you do it and what tools you use, some had better outcomes than others. I believe toxsam has tweeted about an app they have created to work with the cryptoavatars ecosystem a few days back. More importantly. I had like 5-10 people message me over the past 2 years asking me what I was using for YouTube. I am aware of 1-2 who have actually done it by spending 30 minutes researching. Nobody else did. I do not think it is worth going through the trouble of creating an app for so little demand, especially that you can do it with no app. I would much rather have a .vrm avatar export tool that delivers an a vrm file from any wearables combination on demand. That would be really useful for those who want to have some portability of the identity into other worlds. Many people think they want to do something on YouTube and never make it beyond a first video. Because it is hard and because there is not enough of an audience to really justify all the work that goes into it. Peace.

In our app we will incorporate not only a system of shapekeys, we will use the standard of arkit (52 shapekeys) which give us a better fidelity in terms of tracking expressions and a system of shapekeys own (based on shapekeys for AEIOU + extra emotions), compatible with any camera in an executable for mac and windows, the result in an application with a tracking very similar to the result we had in blender but without the technical difficulties involved in its use, in fact the blender file I have it public if you want to review it. But the important thing here besides that are all the possibilities to have in the app during its initial launch and future applications: - Being able to tokenize the backgrounds by making a new category is one of the most favorable points of the dao, in my opinion it means to increase the transactions of the marketplace, with a new category just like this year the emotes or skins. - In the document added, we mentioned that within the roadmap is the possibility of adding another category of dcl avatars created by the community, in that case, if @SWISS makes a tutorial, it will serve for the community to upload their avatars (regardless of whether there is documentation). at that time he could have his avatar as an nft within the decentraland application and not a third party one. Another point that we have in the roadmap is to be able to add a third-party api to be able to generate video calls directly from the app, for now it will only be possible to generate a virtual camera just like in OBS but, we believe that the project has a lot of potential, these last two points we do not add them because it is unrealistic to be able to meet them in 6 months, however they are already contemplated in the general roadmap (among other things). We are aware of the work and the content of swiss and for that reason I can tell you that it is not the same, happy holidays.

Hi @HPrivakos, The proposal is not about how to make a vrm avatar, there are already many tutorials about it, it is about having a specific application for decentraland with the option to use NFTs, being so, at least one of the first vtubing web3 apps, let me explain why this was the best option we came up with: my twitter example first: https://twitter.com/TonyDarko_eth/status/1541371812554199041?s=20&t=0LdItZ8aPVWabG4J82KANg Initially my avatar used a virtual camera in eevee in real time, that means that the lighting, the camera with the focus (depth of field) and many other things could be changed or interact in real time, besides, using eevee the result is aesthetically better, the problem is that it requires some hardware and some knowledge base in blender to use the model correctly. We did a test trying to reduce the steps, using open cv inside blender with my webcam, the result is not bad but it has a delay and consumes too many resources, this is because blender is based on python and the ideal for this kind of things is to have a software based on C++ (just like Unity and Unreal) this is the test we did in blender with a camaraweb https://drive.google.com/file/d/1MTFvC7tDURDQ0KYD_BpobjOw-i0S3KnC/view?usp=sharing